docs(qwen3.6): add data-format ablations and move SciCode diagnosis to ABLATIONS.md - #2835

No actionable comments were generated in the recent review. 🎉

Pre-merge checks: 6

Check name	Status	Explanation
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Security Anti-Patterns		PASS. The pull request changes only three Markdown files under the tutorials directory. It adds no Python code, dependency files, or security-sensitive calls. The added lines contain no `torch.load(..…
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main changes: adding data-format ablations and moving the SciCode diagnosis to ABLATIONS.md.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Full details: Security Anti-Patterns

Explanation

PASS. The pull request changes only three Markdown files under the tutorials directory. It adds no Python code, dependency files, or security-sensitive calls. The added lines contain no torch.load(..., weights_only=False), numpy.load(..., allow_pickle=True), hardcoded trust_remote_code=True, eval(), exec(), or # nosec usage.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.09%. Comparing base (54b684b) to head (6718a04).
⚠️ Report is 16 commits behind head on main.

@@            Coverage Diff             @@
##             main    #2835      +/-   ##
==========================================
- Coverage   69.57%   69.09%   -0.49%     
==========================================
  Files         656      658       +2     
  Lines       73017    76339    +3322     
==========================================
+ Hits        50800    52743    +1943     
- Misses      22217    23596    +1379     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

/claude review

Claude review summary

Scope: Full review. 3 files changed (118 additions, 49 deletions), all Markdown — all three reviewed: the new ABLATIONS.md, the Qwen3.6-35B-A3B/README.md edits, and the tutorials index row. No source, config, or test files in the diff. Since this is docs-only, the review focused on what the prose claims versus what its own tables show, and on whether the things it tells the reader to run actually exist in the repo.

Findings by severity — CRITICAL: 0 · IMPORTANT: 0 · SUGGESTION: 3

All three are internal-consistency issues in the reported numbers. None blocks merge.

1. BF16 SciCode median is reported twice with different values — 1,806 in the data-format table vs 1,817 in the presence-penalty table, for the same build and same stated pooling. 1,817 is the one that reproduces the document's +91.6% headline (3,482 / 1,817 = 1.916).
2. The iteration-500 attribution names the wrong benchmark — "mostly tau2-bench and MMMU-Pro"; the per-benchmark gaps make it tau2-bench (+1.9) and SciCode (+0.7), with MMMU-Pro third at +0.5. Worth fixing because SciCode is the benchmark the rest of the document is about.
3. README compresses "accuracy neutral across four builds" into "accuracy unchanged" next to the single build that moved most (QAD, 40.2 to 39.6). ABLATIONS.md scopes this claim carefully and explicitly declines to attribute that row's drop; the README summary drops the scoping.

Also still open: CodeRabbit's existing finding on line 35 ("5 of 6" should be 4 ahead / 1 tie / 1 behind at iteration 50) — confirmed correct against the table; I did not duplicate it as a separate comment.

What I verified and found correct. The quantitative backbone of this document holds up well:

• Every "Mean delta" cell recomputes exactly from its column (all eight columns, to the stated rounding).
• Cost arithmetic: 735 GPU-h / 128 = 5.7 h, 2,150 / 128 = 16.8 h, 2,150 / 735 = 2.9x, 14 / 4 = 3.5x micro-batches. Tokens-per-step reconciles across all three formats (~16M each), which is the right invariant for a same-tokens cost comparison.
• The fp32 logit figure: 32,768 x 248,320 x 4 B = 30.31 GiB, matching the main README.
• Rate-cost table: 1,475 to 1,192 = 1.24x, 5,109 to 1,067 = 4.8x, −55% tokens.
• Median/mean claims: "flat or slightly up on three of four builds", "mean falls 33-55%", "+71% with the penalty on", "deltas span −0.59 to +0.89", "recovers 94 sub-steps, up to +3.5 pp" — all verified.
• Capped counts: 20/2,704 = 0.7%, 96/2,704 = 3.6%, 338 x 8 = 2,704.
• Self-distillation claims (IFBench fully recovered and the only it50 build to do so; SciCode −1.7; capped rate and median matching chat SFT) all check out against the table.
• Ghost-token control (latest commit): its 4.3% baseline matches the pre-tok it50 capped rate, so it is compared against the right reference.

Repo-facing claims also check out:

• --sft, --sft_hf_dataset, --logit_kl_top_k all exist in examples/megatron_bridge/distill.py; --sft_loss_mode defaults to assistant, consistent with the "loss on assistant tokens only" description.
• The shipped eval_configs/scicode.yaml does carry the presence-penalty line commented out inside params_to_add, exactly as both files claim, with max_new_tokens: 131072 matching the quoted cap.
• The two non-upstreamed options (in-batch packing, zero MoE aux-loss coefficient) are indeed absent from distill.py — the document is correct to say the runs are not reproducible as-is, and flagging that explicitly rather than quietly omitting it is the right call.
• All new in-page and cross-file anchors resolve, including both ABLATIONS.md section slugs and the two README targets.
• No stale references left behind by the removed collapsible block; the README's earlier "see Output length below" pointer still resolves.

Assessment — risk: very low. Documentation only: no code, no API surface, no mode registration, no export path, nothing that can affect a user's checkpoint or runtime. The restructuring into ABLATIONS.md follows the Nemotron-3-Nano tutorial's established layout, and the README's recommended recipe and headline results are unchanged.

Two things worth calling out as good practice: replacing the earlier stop-token probe (one 6K trace, no chat template) with a full-vocabulary probe over 32 sub-steps using the eval's exact prompts is a real strengthening of the evidence, not just a rewording — and it correctly retracts the conclusion the weaker probe supported, redirecting the "next run" advice from top-p / full-vocab KL to on-policy distillation. Documenting a negative result at this level of detail, including the ~2.9x cost of the format that lost, is more useful to the next person than a clean recipe would be.

The three suggestions are worth a quick pass since this file's value rests on its numbers reconciling, but they are cosmetic relative to the content.

🤖 Generated with Claude Code

@cjluo-nv Addressed both items in 89dca1e:

• Data format: the README now says changing the format "did not help here" and marks the comparison as provisional, since the chat path also packs conversations into noisier teacher batches. In ABLATIONS.md, the chat-SFT paragraph now says assistant-only loss "did not remove the runaway in these runs, though chat SFT also changed packing", and points to the caveat.
• Stopping: the bullet now reads "QAD does not suppress ending the reasoning". It says the probe measured </think> on the teacher's sampled text, which is the relevant token because the runaways hit the cap inside <think>, and that the final <|im_end|> was not tested. The closing sentence now notes that, with students dequantized to BF16, this finding does not measure W4A4 serving.

@danielkorzekwa thanks for the review. Addressed in 6718a04:

• megatron_bridge/README.md: says up front that it is the getting-started guide (workflows shown mostly on Qwen3-8B), and one tip now points to tutorials/README.md and names the three tutorials.
• Intro wording: "demonstrating how to quantize both weights and activations" and "settings we did not test and their potential benefits".
• presence_penalty paragraph: rewritten in shorter sentences, same numbers.
• Data format in the data-blend item: cut to one closing clause that keeps the point in context (the lever is what the blend contains; chat-format data did not help, provisionally) with a link to the ablation.
• Step vs iteration: now "~16.8M tokens per iteration (one optimizer step)", plus the token totals at iterations 50 / 300 / 500 (~0.8B / ~5.0B / ~8.4B). The tables keep iteration labels so they line up with the main README's results.
• Verbose results paragraph: the chat-SFT and self-distillation paragraphs now state only the conclusions; the numbers are in the table.
• Confusing chat-SFT section: renamed "How the chat-SFT runs were configured" and opens with "Launch details for the two chat-SFT runs above, not a separate ablation". It also points to Megatron-Bridge: GGML PTQ, QAD and fake-quant export for large MoE and VLM models #2873, which adds the two distill.py flags those runs need.
• 189K responses: generated by the student for the first 190K prompts of the chat-SFT data.